{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 电商移动推荐算法竞赛实践\n",
    "\n",
    "\n",
    "比赛入口\n",
    "\n",
    "* https://tianchi.aliyun.com/competition/entrance/231522/introduction\n",
    "\n",
    "\n",
    "本课件参考了以下链接中的想法，代码为重新编写\n",
    "* https://blog.csdn.net/Datuqiqi/article/details/46834579\n",
    "* https://blog.csdn.net/Snoopy_Yuan/article/category/6924508\n",
    "* https://github.com/chenkkkk/TianChi_YiDongTuiJian_forecast \n",
    "\n",
    "\n",
    "\n",
    "本课件数据下载地址，此数据做了缩短，原来一个月的数据为了方便缩短为12-10到12-18日，如果需要全量数据请自行参加比赛下载，数据说明和之前的天猫品牌行为数据类似，behavior_type分为1浏览，2收藏，3购物车，4购买，多了经纬度数据user_geohash但是没用。\n",
    "* https://pan.baidu.com/s/13SrbmDWGBqQDlnfqB5yI1w\n",
    "\n",
    "\n",
    "重要代码\n",
    "\n",
    "* me.accuracy_score(y_true, y_pred)\n",
    "* me.recall_score(y_true, y_pred)\n",
    "* me.f1_score(y_true, y_pred)\n",
    "* me.precision_score(y_true, y_pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 推荐的原理与特征处理\n",
    "\n",
    "#### 数据读取\n",
    "\n",
    "https://www.bilibili.com/video/BV1s7411w7Xv?p=17\n",
    "\n",
    "#### groupby？\n",
    "\n",
    "https://www.bilibili.com/video/BV1Z5411L7vU?p=9\n",
    "\n",
    "\n",
    "#### unstack\n",
    "\n",
    "https://blog.csdn.net/qq_42874547/article/details/89056000\n",
    "\n",
    "#### 数据处理三板斧——map、apply、applymap详解\n",
    "\n",
    "https://zhuanlan.zhihu.com/p/100064394"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sklearn.metrics as me\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "from sklearn.ensemble import GradientBoostingClassifier #公认这里比较好的算法，我们就直接用了，而且这个方法不太需要特征标准化   \n",
    "from sklearn.model_selection import train_test_split,GridSearchCV "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* 以每天的浏览，收藏，购物车，购买行为数量构建准特征"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "u_i_table=pd.read_csv(r\"F:\\data\\tianchi\\tiancixinrentuijian\\fresh_comp_offline\\tianchi_user2.csv\",parse_dates=[5])\n",
    "i_table=pd.read_csv(r\"F:\\data\\tianchi\\tiancixinrentuijian\\fresh_comp_offline\\tianchi_fresh_comp_train_item.csv\")\n",
    "u_i_table = u_i_table[u_i_table.item_id.isin(i_table.item_id.values)]#过滤掉所有不属于预测商品集的商品行为记录"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead tr th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe thead tr:last-of-type th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>time</th>\n",
       "      <th colspan=\"4\" halign=\"left\">2014-12-10</th>\n",
       "      <th colspan=\"4\" halign=\"left\">2014-12-11</th>\n",
       "      <th colspan=\"2\" halign=\"left\">2014-12-12</th>\n",
       "      <th>...</th>\n",
       "      <th colspan=\"2\" halign=\"left\">2014-12-16</th>\n",
       "      <th colspan=\"4\" halign=\"left\">2014-12-17</th>\n",
       "      <th colspan=\"4\" halign=\"left\">2014-12-18</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>behavior_type</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>...</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>user_id</th>\n",
       "      <th>item_id</th>\n",
       "      <th>item_category</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>492</th>\n",
       "      <th>31040941</th>\n",
       "      <th>6986</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3726</th>\n",
       "      <th>264960</th>\n",
       "      <th>10412</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19137</th>\n",
       "      <th>98921119</th>\n",
       "      <th>3942</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">38745</th>\n",
       "      <th>38770686</th>\n",
       "      <th>3942</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>272985263</th>\n",
       "      <th>3942</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 36 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "time                            2014-12-10          2014-12-11           \\\n",
       "behavior_type                            1  2  3  4          1  2  3  4   \n",
       "user_id item_id   item_category                                           \n",
       "492     31040941  6986                   0  0  0  0          0  0  0  0   \n",
       "3726    264960    10412                  0  0  0  0          2  0  0  0   \n",
       "19137   98921119  3942                   0  0  0  0          0  0  0  0   \n",
       "38745   38770686  3942                   0  0  0  0          0  0  0  0   \n",
       "        272985263 3942                   0  0  0  0          0  0  0  0   \n",
       "\n",
       "time                            2014-12-12     ... 2014-12-16    2014-12-17  \\\n",
       "behavior_type                            1  2  ...          3  4          1   \n",
       "user_id item_id   item_category                ...                            \n",
       "492     31040941  6986                   0  0  ...          0  0          0   \n",
       "3726    264960    10412                  0  0  ...          0  0          0   \n",
       "19137   98921119  3942                   2  0  ...          0  0          0   \n",
       "38745   38770686  3942                   0  0  ...          0  0          0   \n",
       "        272985263 3942                   0  0  ...          0  0          0   \n",
       "\n",
       "time                                     2014-12-18           \n",
       "behavior_type                    2  3  4          1  2  3  4  \n",
       "user_id item_id   item_category                               \n",
       "492     31040941  6986           0  0  0          0  0  0  0  \n",
       "3726    264960    10412          0  0  0          0  0  0  0  \n",
       "19137   98921119  3942           0  0  0          0  0  0  0  \n",
       "38745   38770686  3942           0  0  0          0  0  0  0  \n",
       "        272985263 3942           0  0  0          0  0  0  0  \n",
       "\n",
       "[5 rows x 36 columns]"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "682059"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(u_i_table)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 基于单天行为简单统计的算法\n",
    "\n",
    "* 基于上面的准特征矩阵构建可用的特征矩阵与目标向量，并通过标准流程完成训练，然后基于精准度对模型进行评价\n",
    "* 注意这里以16和17的数据为训练集，以17和18的数据为测试集。\n",
    "\n",
    "* GradientBoostingClassifier()\n",
    "* map(lambda x:1 if x>0 else 0)\n",
    "* me.accuracy_score(y_true, y_pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 二分类的模型评价\n",
    "\n",
    "\n",
    "|  预测与真实 | 真实为正 | 真实为负 |\n",
    "| ------ | ------ | ------ |\n",
    "|预测为正 | TP：True Positive,被判定为正样本，事实上也是正样本。 | FP：False Positive,被判定为正样本，但事实上是负样本。 |\n",
    "| 预测为负| FN：False Negative,被判定为负样本，但事实上是正样本 | TN：True Negative,被判定为负样本，事实上也是负样本 |\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "$\\text{precision} = \\frac{tp}{tp + fp}$\n",
    "\n",
    "* 查准率，所有正预测中正确缺的比率。\n",
    "\n",
    "$\\text{recall} = \\frac{tp}{tp + fn}$\n",
    "\n",
    "* 查全率，所有正样本中被预测到的比率。\n",
    "\n",
    "$F_\\beta = (1 + \\beta^2) \\frac{\\text{precision} \\times \\text{recall}}{\\beta^2 \\text{precision} + \\text{recall}}$\n",
    "\n",
    "* F 值可以解释为 precision （精度）和 recall （召回）的加权调和平均值。 $F_\\beta$ 值达到其最佳值 1 ，其最差分数为 0 。当 $\\beta = 1$, F_\\beta 和 F_1 是等价的， 此时意味着recall和 precision同样重要，# beta值越小，表示越看中precision，beta值越大，表示越看中recall，不同等重要的场景。\n",
    "\n",
    "* 以上指标只关注正样本的预测情况。这和很多实际情况是一致的，比如推荐系统，欺诈识别等等"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "730 268137\n"
     ]
    }
   ],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 样本不均衡问题\n",
    "\n",
    "* 通过随机选择负样本达到正负均衡\n",
    "\n",
    "* pd.concat"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>y</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>user_id</th>\n",
       "      <th>item_id</th>\n",
       "      <th>item_category</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>492</th>\n",
       "      <th>31040941</th>\n",
       "      <th>6986</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3726</th>\n",
       "      <th>264960</th>\n",
       "      <th>10412</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19137</th>\n",
       "      <th>98921119</th>\n",
       "      <th>3942</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">38745</th>\n",
       "      <th>38770686</th>\n",
       "      <th>3942</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>272985263</th>\n",
       "      <th>3942</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                 1  2  3  4  y\n",
       "user_id item_id   item_category               \n",
       "492     31040941  6986           0  0  0  0  0\n",
       "3726    264960    10412          0  0  0  0  0\n",
       "19137   98921119  3942           0  0  0  0  0\n",
       "38745   38770686  3942           0  0  0  0  0\n",
       "        272985263 3942           0  0  0  0  0"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 超参数——调参"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
